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(54) Character segmentation method for vehicle license plate recognition 



(57) A nnethod for segmenting and recognizing 
license plates, in accordance with the present invention 
includes capturing an image of a license plate (100) and 
preprocessing the Image to prepare the image for seg- 
mentation and recognition (110). Forward splitting 
regions of the license plate image into suspected char- 
acters regions, the suspected character regions are rec- 
ognized in the license plate image and a confidence 
score is provided based on a probability of a correct 
match (114). If the suspected characters have a confi- 
dence score below a threshold value, backward merg- 
ing is employed which merges adjacent suspected 
character regions (116). The backward merged sus- 
pected character regions in the license plate are recog- 
nized. If the backward merged suspected character 
regions have a confidence score below the threshold 
value, the backward merging and recognizing steps are 
repeated to improve the confidence score. 
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Description 
BACKGRQUNH 

1. lechnicaLReld 



2. Bes^criptiQiLof th e Related Art 

fhfnl, ThlT'"'" • ^'f "■''^^^'^ P"*"- systems for reading a license plate number have been developed In 

acter rLlnt^n w^^^^ typ.cally consist of three main modules: plate localization, character segmentation and cha^^ 
whih ? T ^ T '^'^'^'^'"^ segmentation is needed to pertbmi character recognition 

7lt J^l^ZT " "lamented characte,^ are not likely to be correctly recogZd In 

ol™? . ^"^-^ the conventional open-loop systems are not due to missing recognition power but 

re^oanZ ""^^ " ^^^^^^^ segmentation is difficuft for license Jlate 

Lnd IkTetc LlTd^:^^^^^^ oharacte^togetherwith dirt, scratches, shad'.^ 

nf mnmr.r ■ ^ ' '**9«<'«s performance of segmentation. In addition, outdoor environment under great variations 
tZT ""^'^ development of a reliable segmentation scheme very complicated 

oSn fOCm'^L^tTr'"?.r"' °" ""'"'y "^^'^ conventional optical character rec- 

tTkJ Thff^T f O*"^'^'- conventional systems perform well for most machine-printed text recognition 
lonlln .h^^T ° ^^-"^ 9^'"- speed of execution. If the characters in license plate are in peS 

frl ""P"^^*"'' °' ^^^'^'^^^ segmentation may be accomplished SrS 

r^lrfe scen'Sr '"^^"^^ '"^^9- usually allS 

Ellumi?«?«?r'." "'""^ ^^3^"^"^^^^^ 
rerent illuminatron and environmental conditions. 

SUMMARY OF TH F IMyjrijTjftM 

m^be imnt!I!!n.!i*°' recognizing license plates, In accordance with the present invention, which 
tTlt '^Pl«7«««» "«"9 a program storage device readable by machine, tangibly embodying a program of instmc- 
tions executable by the machine to perform the method steps, includes capturing an image of a licensHtete and ore 
pr«jess,ng the image to prepare the image for segmentation and recognition. The image 1 fon^aS^p't ir^spect^^^^ 
charactei^ regions of tt,e license plate image. Suspected character regions in the license plate imagVare recognized 
Jlnl T, ^ "^"^ °" ^ P'^''^""^ °' ^ ^^^^'^ the suspected characte j have aTon 

It^Z T'^ " '^^^^^'^ '"^^S'"^ ^^J^^^"t suspected chai^cter regions is perfo^d The 

pSteTl'^Xe'"^^^ " '"^"^^ P'^*^ " the backward meTed sus- 

Sfaretpeated ' ^^^^ "^^"-^"^ ^-9-9 -cognizing 

S!™!L embodiments, which may employ the program storage device, the step of recognizing suspected 

oSrTTd^^eirjr, "^"L"'*' '"^^^ "'^■"'^ ^^^^^^^'^'^^ ^^^-ns with a confidence score b" tow a threth 
old. and the step of forward splrtting regions may include the step of segmenting touching or overlappinq characters in 

^S^of^ c?nTdelcr3 overlapping characters may be recursively applied to achieve the threshold 

^na l Ln^IrTr k" w ^ ° recognizing suspected character regions in the license plate image and pro- 
viding a confidence score based on a probability of a con-ect match may include the step of recognizing characters in 
TnJ.T^^f^^'^ ""^^^ ^'"'"°y'"9 at least one of a neural network and template matching characters m 

[0007] In still other embodiments, the step of recognizing the backward merged suspected character regions In the 
license plate may include the step of recognizing characters in the license plate image by emplo^irj S lei^^ne oTa 
neural network and template matching. The step of preprocessing the image to prepare the imaged segmentation and 
St dr"" "''"^ °' '^'"^"^ '""^ "'^^"^^ P'^*^ 9« ^'tecting edges L StecSeature 

an aoo ! J^^T'"^ ^ °" ^"'^^'"^ ^'^^ P^^j^^^^ion array having a 4hest cos^^o dete^ie 

TJnn t ? ^""^ '^"^'^'"^ "'^""^ P'^'^ accorxJance with the angle of rotation. The step of praX 

the lteense'SLl'H° ^^^"^^"tation and recognition may includ; the steps of refining a posS^f 

the license plate by determining a height of characters on the license plate image by employing a projection function 
computing a spatial variance to determine a center position of the characters, emptoying the clnte^ToSton an^the 
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height of the characters to reduce a size of a region of interest by detecting edges of a plate border and computing a 
length of the edges to identify the region of interest. The step of preprocessing the image to prepare the image for seg- 
mentation and recognition may include the steps of providing a projection profile of pixel intensities across vertical lines 
of pixels in the image, filtering the projection profile and identifying locations of characters in the image depicted by area 
5 below a threshold value in the filtered projection profile. The step of outputting recognized characters of the license 
plate image is preferably performed. The step of comparing recognized characters and character blocks to predeter- 
mined license plate codes and conventions to check accuracy of recognition may be included. 

[0008] These and other objects, features and advantages of the present invention will become apparent from the 
following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompa- 
10 nying drawings. 

BRIEF DESCRIPTION OF DRAWINGS 

[0009] This disclosure will present in detail the following description of preferred embodiments with reference to the 
15 following figures wherein: 

FIG. 1 is a flow diagram showing a license plate recognition method in accordance with the present invention; 
FIG. 2 is a plot of cost versus tilt angle for determining a tilt angle for license plates in accordance with the present 
invention; 

20 FIG. 3 is a plot of S{j) versus pixels for determining character height for license plate characters in accordance with 

the present invention; 

FIG. 4 is a block diagram showing different stages of plate position refinement in accordance with the present 
invention; 

FIG. 5A depicts pixels for overlapped characters to be segmented in accordance with the present invention; 
25 FIG, 5B is a contour projection for determining positions of the characters of FIG. 5A to be segmented in accord- 
ance with the present invention; 

FIG. 5C is a discrimination cost plot for detemriining a location for segmenting the characters of FIG. 5A in accord- 
ance with the present invention; 

FIG. 5D depicts top and bottom profiles for determining heights of the characters of FIG. 5A to be segmented in 
30 accordance with the present invention; 

FIG. 6A depicts pixels for touching characters to be segmented in accordance with the present invention; 
FIG. 6B is a contour projection for determining positions of the characters of FIG. 6A to be segmented in accord- 
ance with the present invention; 

FIG. 6C is a discrimination cost plot for detemriining a location for segmenting the characters of FIG. 6A in accord- 
35 ance with the present invention; 

FIG. 6D depicts top and bottom profiles for determining heights of the characters of FIG. 6A to be segmented in 
accordance with the present invention; 

FIG. 7 is a flow diagram of a recognition based segmentation method in accordance with the present invention; 
FIG. 8 depicts suspected characters regions as determined by a fonward split process in accordance with the 
40 present invention; 

FIG, 9 is a decision tree employed in performed a backward merge in accordance with the present invention; and 
FIGS. 10A and 10B show examples of varying k, a constant for adjusting character boundaries, used in providing 
a higher confidence score in character recognition. 

45 DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

[001 0] The present invention relates to character recognition and more particularly, to a robust method for segment- 
ing characters in a license plate for recognition. Automatic recognition of vehicle license plates is particularly useful for 
high accuracy applications in which car identification is perfomned. Vehicle license plate recognition is beneficial in traf- 

50 fic monitoring and control as well as in maintaining records of through traffic, for example at a parking lot gate. To 
enhance the performance of character segmentation, the present invention performs image preprocessing to deal with 
plate skew and to refine plate position after the image is rotated. This is needed to prepare the data for later processing. 
Vehicle images are degraded due to broken and touching characters. Failure to recognize and segment these charac- 
ters results in the segmentation process being responsible for the major errors experienced by recognition systems. 

55 [0011] The present invention provides a method for more accurately separating touching characters and merging 
broken characters. An efficient and powerful discrimination function is provided by the present invention for identifying 
touching characters based on a differential analysis of character contour distance. The present invention is also sepa- 
rates kerned characters that are overlapped with neighboring characters. 
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IT °^ ^" open-loop operation scheme, the present invention employs a conditionally recursive seamen 

Sln^r^ °' "'^'^^^'^^ ^''^'^P'-. a license pSe w^h I^uSy a^^^^^ 

Jhl t K T"^""^' °' ^""^ ^'■^ inter-related between segmentation and recognitb,^ to JLkTthe oLte 

foZf^rf !f " '^"""'"^ '"3^^"*^' ''"^"^ °" °f t'^^ '^•^^'^'^t^^ recognition 1^006 embodiment S a 

rSttX pli SSf ; "'^^"^^'^ ^^"'^'^ suspldous^character region^ th^rimi : 

r™ffc,. ^ ^ !• 'o^'^led or overlapped. Next, a backward merge procedure is followed to 

r^nfth^^^;^^^^^^^^^^^^^ --^ - p~ -p'- Of ciL^r 

npolS '"'"""'"^ '^""P'^*^ "'^"''''"9 ^"'^ "e^o^ks is realized by the invention This advanta- 

mle bir.Tr' '""^^ segmemation results. The present invention furthertncrd^s 

1 codes wS'^tfolTr'^^^^ '""'"T -9--"^«0" o^the plate. This includes registration reguTi ons 
th«t ^hl ^ ^ appearances (e.g., city/locale codes), alphanumeric fields and character font size Note 

Tln^uT"^ '"'^"^ "'^^"^^ P'^*- charac;Lrr^y not 

^a°r^wL. 1121°""' t*'^* ^'^'"^"ts sliown in FIGS. 1 and 7 may be implemented in various forms of 

. °' combrnatrons thereof. Preferably, these elements are implemented in software on one or more 

^^propnately programmed general purpose digital processing having a processor and memory and inp^output Tr^- 

i?fm!lt. *^^^^TVr ^^^^'^''^ *° ^^'^^'"9^ ^^^^^^'^'^^ ""'"erals identify similar or identfcal 

e ements throughout he several views, and initially to FIG. 1 . a block^low diagram is shown in accoTdance m1 

Tmer^rL I T ' " '"T^ " °' ^^'^^^ a digital camera having a processo aJd 

a memory as descnbed in commonly assigned U.S. application no. (TBD) entitled "LICENSE PLATF RFrrv^Mrrir^M 

Te^nby^l";^^^^^^ 

orelrSJ^'tir^^l°' segmentation preprocessing is perfomied to improve the recognition r^te. Without 
mX.r^'^^ n ^"J segmentation and recognition steps may be difficult. Several preproceS°nq 

efSiu?h . T ^^'^^a"'^'^"- ^'^^'^Pl^. '^^y include image and/or character equalization tn ^0^ 4 

effects such as hght.ng effects (e.g., illumination, distance and/or orientation) need to be accounted for to acSkle a 

Stt^n r 'T"- ^''^'''^ °' '^'■^^ ""P°^"t »° ^l^^ subsequent procesSg isTri^htnTssTor 

rbrrrerarg^raif^^^^^^ 

P017] In block 16, plate tilt orskew is detected and accounted for Plate tilt or skew has a detrimental effect on char- 

^^rtrr p"- °' ^ P'"^" '"^^9^ ^"^^^'^^"g ""Proves the accuraTo segr^ent^^^^^ 

and^ecognrtron. For estrmat.ng the license plate tilt, rts accuracy is dependent on selecting featuri wShe ptte 

S^rt fr^^.r'^? P'«t«« °f tf^e prior art are based on local minima and local maxima pixels 

o^fonlT. ^"^'y^'" °' "P-"""^ ^'^^^'^'^^ °f tl^^ fe^'"^^ Pi'^els through a setTorogonll 

projections detem,.nes a possible tilt angle. However, the accuracy is subject to character background f the plate h^ 

Tcha^rJ tol • ''^r^T ""^ °^ P°'"ts obtained from the top and bottom edges 

r^niL, ? ^ P'^*^ ^'^"^^ P«'«"«' the horizontal dimension. ^ 

thU ■ , ^ ^PP"®'' *° ^" °^ t'^^ character edges and plate edges in the horizontal direction 

The dominant points from edges are then selected by using an adaptive thresh^Sd based on mean and sSndarS' 
a .on. For example, g,ven an angle of 0 degrees rotated wrth respect to center position of the imaged IrtlSproTectbn 
sLl L^h ' « one-dimensional projection ar^y P(e) UsZ^^^^^ 

size with one pixel unit at the sampled resolution due to the small angle rotation, the tm angle is deTJrm ned as followr 

.2;:Th?rrrc(^^^^^^^^^^^^ 



C(e) = £pf(e) 

/=1 



Where P,(G) is the value of the i-th bin found for angle e, and n is the number of bins. The angle that gives the maximum 
be further interpolated through a quadratic fitting function to improve the accuracy, "n ine embodim~mputat" 
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time is reduced by limiting an estimated angle range. The estimate angle range may be limited to within a 10-degree 
inclination so that an exhaustive search can still perform well without degrading speed. If the plate is tilted by a large 
angle an exhaustive search could be time-consuming, a coarse-to-fine search (such as binary or Fibonacci search) can 
be useful to reduce the amount of searching needed to find a maximum. Nonlinear least-squares methods may also be 

5 employed. The tilt angle is accounted to aid in plate character recognition. 

[0020] Referring to FIG. 2, a plot of cost function C(e) (cost) is plotted against an angle of rotation or tilt angle. A 
license plate to be recognized, in this example, is tilted by an angle with respect to the horizontal axis. The plot deter- 
mines that a tilt angle of about 4 degrees is present in the input image of the license plate (e.g., highest cost). This angle 
is then corrected to give an image with a rotation angle of about zero degrees by adjusting the image of the license 

10 plate. 

[0021] in block 18, plate position refinement is perfomned. The output image from license plate localization (blocl< 
1 1 ) usually gives a wider area that includes not only the plate characters but also the plate frame and possibly the body 
of the vehicle. It is necessary to perform a refined localization that leaves out unnecessary regions but retains plate 
characters. The present invention performs the refinement of plate position into vertical and horizontal dimensions by 

15 considering features such as plate border, plate background and plate characters. 

[0022] In general, the best feature to facilitate position refinement is the presence of the plate border on the image 
where it forms a rectangle shape and surrounds with the textural region. By detecting vertical and horizontal edges and 
searching the maximum length of the edges from each side, the refinement can be performed accurately. However, for 
some license plates if one or more plate edges cannot be detected due to a homogeneous background with a vehicle 

20 body, then significant error can result. In these cases, features of plate characters, that is. height and width are meas- 
ured. 

[0023] The present invention estimates the character height in the corrected tilt image. A signature of a stroke tran- 
sition number is employed as a measurement feature corresponding to character existence. Assuming that a plate can- 
didate region has the dimension (pixels) MxN, the projection function SQ) is defined with the summation of all character 
25 signatures in the row: 

SO) = L r(iJl 0 < y < A/ - 1 (2) 

/=0 

30 

where r(i,j) is the number of transitions (object-background or background-object) on the pixel (i,j). This feature is com- 
puted on the binarized image of the plate candidate, i.e., where the suspected license plate image is located. The bina- 
rization is preferably based on a local mean and a local standard deviation overthe image. The threshold at pixel (i,j) is 
35 calculated as: 

T{iJ) = m{iJ) + k-daj) (3) 

where m(i,j) and d(i,j) are the sample mean and standard deviation values in a local neighborhood of (i,j) respectively. 

40 In a preferred embodiment, a 16x16 neighborhood is to preserve local details and minimize noise effects. The value of 
k Is used to adjust how much of the character boundary is taken as a part of the given image. For example, k = -0.2 to 
give well-separated print characters. It should be noted that a constant k may result in width variation between charac- 
ters due to non-uniform illumination. The effect of k corresponding to a recognition result will be illustrated below. Sta- 
tistically, the average transition number per row of textural region is much higher than that of non-textural regions. If S(j) 

45 is segmented properly, the result of maximum spread distance will correspond to character heights. 

[0024] Referring to FIG. 3, a projection function SQ) is shown where the height of the characters is the distance from 
the values of S(j) above a threshold. The threshold of S(j) is obtained adaptively based on a mean value and a standard 
deviation of S(j). Once the height of characters is determined, the region of interest in the vertical dimension is further 
extracted - 

so [0025] The width of plate numbers may be estimated based on prior knowledge of the ratio of plate width to plate 
height, and only the central position of plate number in the horizontal axis needs to be determined. A spatial variance 
method may be implemented to estimate this position. If the spatial variance is computed along a line of the license 
plate (e.g., horizontal), regions with high spatial variance correspond to text regions, and the regions with low spatial 
variance correspond to non-text regions. Thus, the position associated with the highest variance can be selected as the 

55 center position of the plate number on the horizontal axis. 

[0026] With the analysis of plate edges, the accuracy of refinement can be improved and spurious detection can be 
eliminated. After refinement is performed along the horizontal axis, the region of interest is extracted again then resa- 
mpled to a fixed resolution for the subsequent segmentation and classification. FIG. 4 shows the result for plate position 



5 

JSDOCID: <EP 1085456A2J_> 



10 



75 



20 



EP 1 085 456 A2 

refinement at different stages of the process. 

iStinn 2° Character Segmentation is performed. This includes identifying characters in the license plate by 

Tck 22 em^o!, '''^.^''^'^''^'^ recognition. (See also block 1 1 0 of FIG. 7). In accordance wrth the invention! 
formed if hTpS segmentation. In block 22. segmentation of touching/overlapping characters is per^ 

OCm inL«t T"^ ,1 "t^'"^"^^*'"" °f ^""^^''"a ^f^^'^'^ters IS of importance to any optical character recognition 
(OCR) applications. Several techniques have been developed in the prior art which suffer an inability to discriminate 

S,.. SJT'^'lf?" °" ^^P'°y«^ invention. Considering a touching 

character with the width W, a contour projection function Vc(k) is defined as: 



WcCf) = bottom(k) - top{k), k = 1,2,3 W 



(4) 



where the function top(k) gives a top position in the vertical axis as seen from the top profile of the external contour of 

Irof r r ?• ''°"°'^<'> 9-^"" " ^"^^ -een from the boSom 

profile Of the external contour of the touching characters. See FIGS. 5D and 6D for top and bottom profiles for the 
kerned cha^cter M 7" in FIG. 5A and the touching characteiB "48" in FIG. 6A. respectively. Using the above pJo^^on 
function with several adjacent columns, the segmentation discrimination function. F,(k). is then defined as foZs 



''cW - L [ vjk) J ■ d<k<w.d (5) 



cf=n1 



30 



35 



7cZ JilZ M k7 P°«tion to adjacent character posftions. The summation interval from n1 

to n2 denotes all possible offsets detemiined empirically based on the size of the characters and the overlapped con- 
dition m the text. Given the robustness fbrthe present invention, n1=l and n2=2 was found to resultfn ™S discrim- 
.r^tror, capabil,^. Other ^lues for n1 and n2 are also contemplated. FIGS. 5B and 6B show contour p^nV v7k) 
^ a function o k for FK3S. 5A and 6A, respectively FIGS. 5C and 6C illustrate the values of the discriminition function 
Fe(k) as a function of k. Each offset d contributes to the values of the discrimination function with significant increases 
^t wh ch tol/tT""' *'i? TVr- ^"'"^ ' -"didate-cutfing points are sefectX ~ 

Li? ^ 1^ " ^^""^ P~P°^«d '=°"tour projection is only applied for suspicious char 

a^^rs that failed dunng classffication. It is redundant to check those segmented characters ifthey can bercognSd 

foJ?d^nSn? K "'^ K '^""^""^ P^°j^*°" -^^y be combined with pixel profection 

or rdentrtying touching characters that mix wrth lower case characters in machine-printed documents 

Sition?''°T"'°'"''^^^'' segmentation of block 22 is performed in accordance with the present invention. The 
o^ln *"i«9e for character components that match classes in its ^phabet Rec 

ognrtron compares templates or employs a statistical classrfier to identHy characters and return a confidence sco^ 
based on a probability that the character has been properly identified. 
40 [0030] The principle of operation of the present invention is quite different from open-loop segmentations based on 
Sjeote^ntaZ "^"^ '^'"^'^ '^''"P'^*^ """^ character recognition. Recognln 

L^nrrt «1 r "^^ ^ '""""^ ^'"•^"^ °' ^^'^'"^ P^^**^^ sequences of candidate segmen- 

tetons that are confirmed by character recognition. The input image can generate multiple sequences by varying the 

« rntln r.nri '"'"'"'^^ ^^^^ ^'^^^^^^ confidence measure is chosen ^ a slg 

45 rnentation result. However, the problem with the prior art methods is that it is drfficult to choose a suitable window on 
the initia tnaL particularly for proportionally spaced characters. Tsujimoto and Asada. in "Resolving ambiguity in seq- 

?hlv rn .v^H^^H K ^ segmentation-and-recognition algorithm for segmenting touching chai^cters 

t?r^f.oh c« K ^ '"""^'^^ ""'^^ t^"t^tive break posrtions. and then confimis these break positions 

ntT"? ^^^^'^^^'"Sa decision tree constructed by recognition results. Unfortunately, their work could be computetionaffy 
intensive if more break positions are involved for building a decision tree ipuranonaiiy 

Kments^fo^i^^JtJiillH accordance wrth the invention, a conditionally recursive segmentation method that 
Z ^It^^ ^ and backward merge procedure and is based on the output of recognition is shown. In block 

bici iTt.S r P^^P^f f ^^'"9 P^'^^^'"^'^ °" !"^age for example a license plate image as described above. In 
th^cJn elimir^l m""^ kI' P"^°^7«V'' '"P"' ""^^^ segmented into regions based on vertteal projection 
nrl^ T T / ""^^""^ ^ ^ ^"'f" conventional black-pixel proj«:tion the 

oZ nr " '^'"^""3 '^^^^^ P'""' V^^'^^' P^°j«^«°n '"'^'"des applying a low 

pass filter to the projection profile. Then, the distinguished positions between characters are determined through sub- 
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traction of original projection function and fittered function. The details are illustrated in comnnonly assigned U.S. appli- 
cation no. (TBD). entitled "LICENSE PLATE RECOGNITION WITH AN INTELLIGENT CAMERA." previously 
incorporated herein by reference. The recognition engine is preferably employed as a check. If the segmented image 
includes a confidence score below a threshold, the process is repeated after suspected characters are split or com> 
5 bined. 

[0032] After initial segmentation, each resulted region (i.e., suspected or potential character region) can be catego- 
rized as either a broken character, a single character, a touching character or an unknown symbol. If the license plate 
has a high quality image, almost every segmented region can be classified with a high confidence measure that indi- 
cates a recognized character at this stage. However, most license plate images are not in such a condition. If the 
10 regions result in unrecognized characters, each of them will be checked by component analysis to filter out small com- 
ponents such as screws or bolts present in the image of the license plate. 

[0033] In block 114, the described discrimination function for separating touching characters is then applied to 
those over sized regions to see whether they can be further split. It is practical to assume that the associated cost of 
discrimination function is higher than the defined threshold so that unnecessary splitting can be avoided. The forward 
15 split procedure of block 1 14 is immediately followed by recursive recognition which returns to block 114 until no break 
point is found in suspicious regions where a low confidence score is obtained for recognizing the character in question. 
An example is shown in FIG. 8 wherein six character regions (1-6), as indicated between arrows, denote positions at 
which to split the character image. 

[0034] In block 116, a backward merge process is performed to evaluate regions in the image to determine if a 
20 higher confidence is achievable by combining regions. This is a recursive recognition process as indicated by the feed- 
back loop in FIG. 7. Referring to FIG. 9, each node 200 for each suspected character region 1-6 (see FIG. 8) might form 
an actual character depending on the recognition of the region. When a node can be classified with a significant confi- 
dence measure, the result is stored in the relevant node. Then, the subsequent nodes that are the possible combina- 
tions of the remaining regions are processed. Nodes that give poor recognition are not evaluated further (e.g., nodes 
25 with * in FIG. 8). Each subsequent evaluation is constrained to only the neighborhood in the region in question and its 
2 contiguous adjacent nodes. This reduces computational time. The tree is constructed until no regions are left. The 
most likely path through the tree will be selected as the correct segmentation, and the corresponding recognition will be 
immediately available. If none of the paths result in a good recognition, the less likely branches can be evaluated. For 
example, considering FIG. 8, region 1 is determined in node 200a not to include a recognized character (an * is placed 
30 in the node). The process then considers regions 1 and 2 in node 200b. An "N" is recognized with sufficient confidence. 
In node 200c. regions 1 , 2 and 3 are checked, but the evaluation does not yield results with a sufficient confidence level. 
Processing continues as shown. 

[0035] In block 115, character recognition is performed in accordance with the present invention. Character recog- 
nition may be performed in a plurality of different ways. In particularly useful embodiments, a neural network, template 
35 matching or a combination of both may be employed. Template matching will now be described In greater detail. Neural 
network implementation is described in greater detail in U.S. application no. (TBD). "LICENSE PLATE RECOGNITION 
WITH AN INTELLIGENT CAMERA", previously incorporated herein by reference. 

[0036] The segmentation method of the present invention depends for its accuracy on the character recognition 
method that is applied to the split regions. It is independent of that algorithm in the sense that any recognition methods 

40 may be employed with the segmentation. Therefore, in addition to neural network serving as an OCR engine, the 
present invention implements template matching to further facilitate the recognition task. Template matching applies to 
segmented regions enclosed by rectangles with connected components in the regions having an average size. Further, 
the regions (with a character or portion thereof) are recognized by the neural network with a lower confidence measure 
(below a threshold). In such case, the region of interest is first binarized and scaled to the same size of the templates 

45 in a database, for example, 15x25 or 20x30 pixels. A nomiallzed matching index with the range of -1 and 1 is defined 
as the confidence measure obtained by pixel-to-pixel comparison between the saved symbol image and the input object 
image. As the confidence measure approaches 1, the saved images (virtual samples) associated with character sym- 
bols are classified, which implies a perfect match (confidence of 1) that the class of the input image is the same as that 
of the saved one, and vice versa. A threshold of about 0.5 may be chosen to filter out non-match characters. The 

so threshold may be adjusted according to the application. Template matching provides for an exhaustive search in a 
stored database including a variety of symbol images, including different sizes and styles of the same character, various 
illumination effects, shadows, blunred images, etc. 

[0037] In the matching process, it should be noted that background pixels that match are usually not counted, since 
that strongly biases the matching index. However, if a hole is considered as a structure, the set of background pixels in 
55 the hole is permitted as a character. To do this, the holes are marked with a pixel value other than those used for back- 
ground and object pixels. Then, an extra case (I.e., hole pixels) is combined for the matching. In essence, hole pixels 
are treated in the same way as object pixels, but are a distinct class and do not match with an object pixel. 
[0038] Niblack's method maybe adapted for binarization of images based on the local mean and local standard 
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deviation over the image. (See e.g., O.Q Trier, et al., "Goal-directed evaluation of binarization methods", IEEE Transac- 
tion on Pattern Analysis and Machine Intelligence, Vol. 17 , No. 12, pp. 1 191-1201, December 1995). In Eq. 3, the value 
of k is used to adjust how much of the character boundary is taken as a part of the given image. As image environment 
changes, character widths are likely to be varied to different levels. A constant k may not be suitable to give a good 

5 matching. Therefore, a multi-threshold is derived based on the variation of k. In one embodiment, if k is incremented 
from -0.5 to 0.5 at an interval of 0.25 a good selection that covers a large variation in symbols of the data set. FIGS. 
10A and 10B show how different thresholds affect character boundary. Image 400 and 410 are the image to be 
matched. By varying k a better match may be obtained for each image. Image 400 has a highest confidence when k = 
0.5 in FIG. 10A, and image 410 has a highest confidence when k= 0.0 in FIG. 108. 

10 [0039] Experiments were carried out by the inventors in accordance with the present invention employing an intel- 
ligent camera, as described In U.S. application no. (TBD), "LICENSE PLATE RECOGNITION WITH AN INTELLIGENT 
CAMERA", previously incorporated herein by reference. The system performance was based on overall recognition rate 
of entire license plates under varying distances and illumination effects. Illumination changes were performed by vary- 
ing the lens aperture with f number 2, 2.4, and 4. It should be noted that the recognition rate defined here is in terms of 

15 number of license plates rather than number of characters. 

[0040] A closed-loop of the recognition and segmentation stage increases the robustness for the effects of environ- 
mental changes. Although the present invention is demonstrated using German license plates, the algorithms are gen- 
eral and extensible for different nationalities of license plate as well as to other character recognition tasks. 
[0041] In the automated license plate recognition system many reading errors are caused by inadequate character 

20 segmentation. In particular, character segmentation becomes difficult as the acquired vehicle images are seriously 
degraded. The present Invention utilizes computer vision techniques and proposes a recognition-based segmentation 
method coupled with template matching and a neural network. This enhances the accuracy of the recognition system 
that alms to automatically read license plates. Algorithmic improvements for a projection-based segmentation are 
described here. In the preprocessing stage, plate skew detection and position refinement methods are developed to 

25 prepare the data for later process. For separating touching characters a discrimination function is presented based on 
a differential analysis of character contour distance. A conditionally recursive segmentation with the feedback of recog- 
nition is developed for effectively splitting touching characters and merging broken characters. 

[0042] Having described preferred embodiments for character segmentation method for vehicle license plate rec- 
ognition (which are intended to be illustrative and not limiting), it is noted that modifications and variations can be made 
30 by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made 
in the particular embodiments of the invention disclosed which are within the scope and spirit of the invention as out- 
lined by the appended claims. Having thus described the invention with the details and particularity required by the pat- 
ent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims. 

35 Claims 

1- A method for segmenting and recognizing license plates comprising the steps of: 

capturing an image of a license plate; 
"^0 preprocessing the image to prepare the image for segmentation and recognition; 

forward splitting regions of the license plate image into suspected characters regions; 

recognizing suspected character regions In the license plate image and providing a confidence score based on 
a probability of a correct match; 

if the suspected characters have a confidence score below a threshold value, backward merging adjacent sus- 
45 pected character regions; 

recognizing the backward merged suspected character regions in the license plate; and 

if the backward merged suspected character regions have a confidence score below the threshold value, 

repeating the backward merging and recognizing steps. 

50 2. The method as recited in claim 1 , wherein the step of recognizing suspected character regions in the license plate 
Image includes determining regions with a confidence score below a threshold, and the step of forward splitting 
regions includes the step of segmenting touching or overiapping characters in adjacent suspected character 
regions by employing a discrimination function based on contour projections of the characters. 

55 3. The method as recited in claim 2, wherein the step of segmenting touching or overlapping characters is recursively 
applied to achieve the threshold value of a confidence score. 

4. The method as recited in claim 1 , wherein the step of recognizing suspected character regions in the license plate 
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image and providing a confidence score based on a probability of a con-ect match includes the step of recognizing 
characters in the license plate image by employing at least one of a neural network and template matching. 

5. The method as recited in claim 1, wherein the step of recognizing the backward merged suspected character 
5 regions in the license plate includes the step of recognizing characters in the license plate image by employing at 

least one of a neural network and template matching. 

6. The method as recited in claim 1 , wherein the step of preprocessing the image to prepare the image for segmen- 
tation and recognition includes the steps of: 

10 

detecting pretilt in the license plate image by detecting edges and selecting feature points to determine a pro- 
jection array on the edges; 

selecting the projection array having a highest cost to determine an angle of rotation; and 
correcting the license plate image in accordance with the angle of rotation. 

15 

7. The method as recited in claim 1 , wherein the step of preprocessing the image to prepare the image for segmen- 
tation and recognition includes the steps of: 

refining a position of the license plate by: 

20 

determining a height of characters on the license plate image by employing a projection function; 
computing a spatial variance to determine a center position of the characters; 

employing the center position and the height of the characters to reduce a size of a region of interest by 
detecting edges of a ptate border and computing a length of the edges to identify the region of interest. 

25 

8. The method as recited in claim 1 , wherein the step of preprocessing the image to prepare the image for segmen- 
tation and recognition includes the steps of 

providing a projection profile of pixel intensities across vertical lines of pixels in the image; 
30 filtering the projection profile; and 

identifying locations of characters in the image depicted by area below a threshold value in the filtered projec- 
tion profile. 

9. The method as recited in claim 1 , further comprising the step of outputting recognized characters of the license 
35 ptate image. 

10. The method as recited in claim 1 , further comprising the step of comparing recognized characters and character 
blocks to predetermined license plate codes and conventions to check accuracy of recognition. 

40 11. A method for segmenting and recognizing license plates comprising the steps of: 

capturing an image of a license plate; 

preprocessing the image to prepare the image for segmentation and recognition; 

forward splitting regions of the license plate image into suspected characters regions by segmenting touching 
45 or overlapping characters in adjacent suspected character regions by employing a discrimination function 

based on contour projections of the characters, when the suspected characters have a confidence score below 
a threshold value,; 

recognizing suspected character regions in the license plate image and providing a confidence score based on 
a probability of a correct match; 
50 if the suspected characters have a confidence score below a threshold value other than due to touching or 

overlapping characters, backward merging adjacent suspected character regions; 
recognizing the backward merged suspected character regions in the license plate; 

if the backward merged suspected character regions have a confidence score below the threshold value, 
repeating the backward merging and recognizing steps; 
55 comparing recognized characters and character blocks to predetermined license plate codes and conventions 

to check accuracy of recognition; and 
outputting recognized characters of the license plate image. . 
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12. The method as recited in claim II , wherein the step of segmenting touching or overlapping characters is recursively 
applied to achieve the threshold value of a confidence score. 

13. The method as recited in claim 1 1 , wherein the step of recognizing suspected character regions in the license plate 
5 image and providing a confidence score based on a probability of a correct match includes the step of recognizing 

characters in the license plate Image by employing at least one of a neural network and template matching. 

14. The method as recited in claim 11, wherein the step of recognizing the backward merged suspected character 
regionis in the license plate includes the step of recognizing characters in the license plate image by employing at 

10 least one of a neural network and template matching. 

15. The method as recited In claim 1 1 , wherein the step of preprocessing the image to prepare the image for segmen- 
tation and recognition includes the steps of: 

15 detecting pretilt in the license plate image by detecting edges and selecting feature points to determine a pro- 

jection array on the edges; 

selecting the projection array having a highest cost to detemriine an angle of rotation; and 
correcting the license plate image in accordance with the angle of rotation. 

20 16. The method as recited in claim 1 1 , wherein the step of preprocessing the image to prepare the image for segmen- 
tation and recognition includes the steps of: 

refining a position of the license plate by: 

25 detennining a height of characters on the license plate image by employing a projection function; 

computing a spatial variance to determine a center position of the characters; 

employing the center position and the height of the characters to reduce a size of a region of interest by 
detecting edges of a plate border and computing a length of the edges to identify the region of interest. 

30 17. The method as recited in claim 1 1 , wherein the step of preprocessing the image to prepare the image for segmen- 
tation and recognition includes the steps of 

providing a projection profile of pixel intensities across vertical lines of pixels in the image; . 
filtering the projection profile; and identifying locations of characters In the image depicted by area below a 
35 threshold value in the filtered projection profile. 

18. A program storage device readable by machine, tangibly embodying a program of instructions executable by the 
machine to perform method steps for segmenting and recognizing license plates, the method steps comprising: 

40 capturing an image of a license plate; 

preprocessing the image to prepare the image for segmentation and recognition; 
forward splitting regions of the license plate image into suspected characters regions; 

recognizing suspected character regions in the license plate image and providing a confidence score based on 
a probability of a correct match; 
45 if the suspected characters have a confidence score below a threshold value, backward merging adjacent sus- 

pected character regions; 

recognizing the backward mergecl suspected character regions in the license plate; and 

if the backward merged suspected character regions have a confidence score below the threshold value, 

repeating the backward merging and recognizing steps. 

50 

19. The program storage device as recited in claim 1 8, wherein the step of recognizing suspected character regions in 
the license plate image includes determining regions with a confidence score below a threshold, and the step of 
forward splitting regions includes the step of segmenting touching or overlapping characters in adjacent suspected 
character regions by employing a discrimination function based on contour projections of the characters. 

55 

20. The program storage device as recited in claim 19, wherein the step of segmenting touching or overlapping char- 
acters Is recursively applied to achieve the threshold value of a confidence score. 
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21. The program storage device as recited in claim 1 8, wherein the step of recognizing suspected character regions in 
the license plate image and providing a confidence score based on a probability of a correct match includes the 
step of recognizing characters in the license plate image by employing at least one of a neural network and tem- 
plate matching. 

5 

22. The program storage device as recited in claim 18, wherein the step of recognizing the backward merged sus- 
pected character regions in the license plate includes the step of recognizing characters in the license plate image 
by employing at least one of a neural network and template matching. 

10 23. The program storage device as recited in claim 1 8. wherein the step of preprocessing the image to prepare the 
image tor segmentation and recognition includes the steps of: 

detecting pretilt in the license plate image by detecting edges and selecting feature points to determine a pro- 
jection array on the edges; 
IS selecting the projection array having a highest cost to determine an angle of rotation; and 

correcting the license plate image in accordance with the angle of rotation. 

24. The program storage device as recited in claim IB, wherein the step of preprocessing the image to prepare the 
image tor segmentation and recognition includes the steps of: 

20 

refinirg a positon of the license plate by: 

detetrining a height of characters on the license plate image by employing a projection function; 
computng a spatial variance to determine a center position of the characters; 
25 emp oymg the center position and the height of the characters to reduce a size of a region of interest. 

25. The program storage device as recited in claim 18, wherein the step of preprocessing the image to prepare the 
image for segmentation and recognition includes the steps of 

30 provioing a projection profile of pixel intensities across vertical lines of pixels in the image; 

filtering ne projection profile; and identifying locations of characters in the image depicted by area below a 
threshoks value in the filtered projection profile. 

26. The program storage device as recited in claim 1 8, further comprising the step of outputting recognized characters 
35 of the license plate image. 

27. The program storage oevice as recited in claim 1 8, further comprising the step of comparing recognized characters 
and character blocks to predetemnined license plate codes and conventions to check accuracy of recognition. 
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